docker0网桥异常

异常现象

# #flanneld-start[1853]: I0321 16:55:46.027824    1853 network.go:243] L3 miss but route for 172.30.41.2 not found

异常原因

根据分析产生异常原因是flanneld node的网段在etcd 中没有配置好，当与别的node交互时查询目标node ip时失败，因为etcd中没有保存节点子网ip段信息，或者是保存ip段信息是错误的，可以手动添加ip段，ip段 指的是Flannel 网卡的IP地址

查看docker的Flannel配置是否生效

systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/docker.service.d
           └─flannel.conf
   Active: active (running) since 三 2018-04-25 13:19:52 CST; 12min ago
     Docs: https://docs.docker.com
 Main PID: 24187 (dockerd)
   Memory: 85.4M
   CGroup: /system.slice/docker.service
           ├─24187 /usr/bin/dockerd --exec-opt native.cgroupdriver=systemd --bip=172.30.17.1/24 --ip-masq=true --mtu=1450 -s=overlay2 --log-driver=json-file

查看flannel进程状态

systemctl status flanneld -l确认没有错误信息，如果存在 异常现象 信息时就查看ectd中的subnet数据，因为podip跨主机与容器通信时会先去etcd中查找到对的IP段，再查找到flannel的网卡mac地址和IP地址

etcd中查看子网信息

etcdctl --endpoints=https://x.x.x.x:2379,https://x.x.x.x:2379 \
> --ca-file=/etc/kubernetes/ssl/ca.pem \
> --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
> --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \
> ls /kube-centos/network/subnets

subnets信息如下:
/kube-centos/network/subnets/172.30.17.0-24
/kube-centos/network/subnets/172.30.69.0-24
/kube-centos/network/subnets/172.30.95.0-24

etcd查看节点信息

etcdctl --endpoints=https://x.x.x.x:2379,https://x.x.x.x:2379 \
> --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem \
>  --key-file=/etc/kubernetes/ssl/kubernetes-key.pem \ 
> get /kube-centos/network/subnets/172.30.41.0-24

节点信息:
{"PublicIP":"x.x.x.x","BackendType":"vxlan","BackendData":{"VtepMAC":"36:f8:86:0e:bf:d1"}}
publicip是Flannel指定网卡地址<资源配置实践－Flannel指定网卡通信>目录中有说明
backendData是flannel.1的mac地址

etcd配置节点信息

etcdctl --endpoints=https://x.x.x.x:2379,https://x.x.x.x:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem set /kube-centos/network/subnets/172.30.17.0-24 '{"PublicIP":"10.27.70.6","BackendType":"vxlan","BackendData":{"VtepMAC":"b2:34:47:0b:b3:0a"}}'
# 把所需节点的subnet配置完后把旧的subnet记录删除
etcdctl --endpoints=https://x.x.x.x:2379,https://x.x.x.x:2379 --ca-file=/etc/kubernetes/ssl/ca.pem --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem rm /kube-centos/network/subnets/172.30.64.0-24

重启Flannel

systemctl restart flanneld
systemctl status flanneld

解决方案

自己手动添加subnets信息，或者指docker的子网网段信息(BIP,IPMASQ,MTU)等信息指定就OK